setup

source rmacroRDM functions

First source the rmacroRDM functions. Currently the best way is to just source from github using RCulr:getURL().

WARNING: Under continuous development

require(RCurl)
eval(parse(text = getURL("https://raw.githubusercontent.com/annakrystalli/rmacroRDM/master/R/functions.R", ssl.verifypeer = FALSE)))
eval(parse(text = getURL("https://raw.githubusercontent.com/annakrystalli/rmacroRDM/master/R/wideData_function.R", ssl.verifypeer = FALSE)))

setup file.system

Next, to initialise the project we need to supply valid pathways to project folders containing:

  • data: folder containg private input and output folders are stored, usually a googledrive folder) and
  • code: folder containing scripts associated with the project. This is usually an RStudio project directory, ideally version controlled on github.

The function sets those directories in the global environment.

setDirectories(script.folder = "~/Documents/workflows/Brain_size_evolution/", 
               data.folder = "~/Google Drive/Brain evolution/",
               envir = globalenv())

Once project folders have been set, we set up the file system by creating the required folders (if they don’t exist already) in the project folders.

setupFileSystem(script.folder = "~/Documents/workflows/Brain_size_evolution/", 
                data.folder = "~/Google Drive/Brain evolution/")

initialise database configurations and attach to search pathway

In this step, we initialise the environment with some required parameters to build the database and process files to it. The call below shows the default initialisation settings which you would get if you just called inti_db()

init_db(var.vars = c("var", "value", "data.ID"),
                    match.vars = c("synonyms", "data.status"),
                    meta.vars = c("qc", "observer", "ref", "n", "notes"),
                    taxo.vars = c("genus", "family", "order"),
                    spp.list_src = NULL)
init_db(spp.list_src = "D0")
configuring: 
master.vars
match.vars
meta.vars
spp.list_src
taxo.vars
var.vars
NULL

The function appends the objects specified in the function to environment master_config at position 2 in the search path (note position of GlobalEnvironment = 1).

Here’s a list of the values of the objects we just attached as configurations:

setup input.folder

Next we setup the folders in input.folder/pre/ and post/ according to the configurations set in the previous step.

If the correct setup already exists, no action is taken:

setupInputFolder(input.folder)
dirs <- list.dirs(paste(input.folder, "pre/", sep = ""), full.names = T)

print(dirs)
[1] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre/"         
[2] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//csv"     
[3] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//n"       
[4] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//notes"   
[5] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//observer"
[6] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//qc"      
[7] "/Users/Anna/Google Drive/Brain evolution/inputs/data/pre//ref"     



populate file.system

The functions take advantage of the structure of the file.sytem to automate loading and linking of data and metadata through appropriate naming and location of files within the file.system.

NB meta.var data sheets should be named with the same name as the data data sheet. Take care during this stage to ensure files are named correctly and stored in the appropriate folders.

load required data

make fcodes (folder codes) vector

The fcodes vector specifies the details of folders in pre/ and post/ input.folder folders. It also creates appropriate code prefixes for each type of data or meta.var sheet. Note that “D” is reserved for data files, “R” for ref files and “N” for n files.

fcodes <- ensure_fcodes(meta.vars)
print(fcodes)
         D          R          N          Q          O         NO 
     "csv"      "ref"        "n"       "qc" "observer"    "notes" 

set file.names vector

Specify the file.names of the files you wish to process. If you want to process all files in the file.system use file.names = NULL.

file.names <- create_file.names(file.names = c("brainmain2.csv", 
                                  "Amniote_Database_Aug_2015.csv", "anagedatasetf.csv"))
dcodes succesfully extracted from 'data_log.csv'
print(file.names)
                             x0                              x1 
               "brainmain2.csv" "Amniote_Database_Aug_2015.csv" 
                             x2 
            "anagedatasetf.csv" 

load system reference files

load_sys.ref(fileEncoding = "mac", view = F)
loading: 
metadata
data_log
vnames
attaching to env: sys.ref
NULL

*use view = T to open a viewer for each of the sys.ref files

metadata.csv

kable(head(metadata, 10))
code cat descr scores levels type units
species NOMINAL Scientific name NA NA NA NA
brain.volume MORPHOLOGICAL Brain volume NA NA CON mm
brain.mass MORPHOLOGICAL Brain mass NA NA CON g
male.brain.mass MORPHOLOGICAL Male brain mass NA NA CON g
female.brain.mass MORPHOLOGICAL Female brain mass NA NA CON g
body.mass MORPHOLOGICAL Body mass NA NA CON g
male.body.mass MORPHOLOGICAL Male body mass NA NA CON g
female.body.mass MORPHOLOGICAL Female body mass NA NA CON g
telencephalic.volume.fraction MORPHOLOGICAL Telencephalic volume fraction NA NA CON mm
female.maturity LIFE HISTORY TRAIT Female maturity NA NA INT d

data_log.csv

kable(data_log)
dcode file.name descr source source.contact method notes
D0 brainmain2.csv NA NA NA NA NA
D1 Amniote_Database_Aug_2015.csv NA NA NA NA NA
D2 anagedatasetf.csv NA NA NA NA NA
D3 lifehistraits.csv NA NA NA NA NA
D4 parentalcare.csv NA NA NA NA NA

vnames.csv

kable(head(vnames, 10))
code D0 D1 D2 D3 D4 R1 N1
species Scientific name species Scientific.name Scientific name Scientific_name species species
genus NA genus NA NA NA genus genus
brain.volume_n n NA Sample.size NA NA NA NA
brain.volume Brain Vol. NA NA NA NA NA NA
brain.volume_ref Source brain vol. NA NA NA NA NA NA
brain.mass Brain mass (g) NA NA NA NA NA NA
brain.mass_n n brain mass NA NA NA NA NA NA
brain.mass_ref source brain mass NA NA NA NA NA NA
male.brain.mass male brain mass NA NA NA NA NA NA
male.brain.mass_n male brain mass no NA NA NA NA NA NA

load syn.links.csv

Used for taxonomic matching (plans to automate this by integrating package taxize. supplied syn.links only relates to birds).

syn.links <- read.csv(text=getURL("https://raw.githubusercontent.com/annakrystalli/rmacroRDM/master/data/input/taxo/syn.links.csv", 
                                  ssl.verifypeer = FALSE), header=T)



process file.system

This step processes the .csv copies of the raw data in the pre/ folder and writes the processed files as csvs to the post/ folder, ready to be matched. The processing conserves the file.names throughout. It is important for this step that the file.system is correctly populated (ie. meta.var data sheets should be named with the same name as the data data sheet and in the correct folder).


The function runs a basic processing stage for each file in file.names:


custom processing scripts

you can include extra processing scripts for individual files by adding them into the {script.folder}process/ folder. To be loaded correctly, scripts need to be named appropriately:

process_file.system(file.names, fcodes)
[1] "all files in file system have valid vnames columns"
===================================================================
processing Amniote_Database_Aug_2015.csv D1 
[1] "source complete"
sourced: process/Amniote_Database_Aug_2015.R
'genus_species' combined in species column. Genus data removed 
df nrow (species) : 9802 df ncol (traits) : 20 
*** 
===================================================================
processing anagedatasetf.csv D2 
df nrow (species) : 1191 df ncol (traits) : 18 
*** 
===================================================================
processing brainmain2.csv D0 
duplicate species name in D0
df nrow (species) : 2586 df ncol (traits) : 24 
*** 
===================================================================
processing Amniote_Database_Aug_2015.csv R1 
[1] "source complete"
sourced: process/Amniote_Database_Aug_2015.R
'genus_species' combined in species column. Genus data removed 
df nrow (species) : 9802 df ncol (traits) : 20 
*** 
===================================================================
processing Amniote_Database_Aug_2015.csv N1 
[1] "source complete"
sourced: process/Amniote_Database_Aug_2015.R
'genus_species' combined in species column. Genus data removed 
df nrow (species) : 9802 df ncol (traits) : 19 
*** 



Create database

create spp.list

Use spp.list_source to specify dcode of file.name to extract spp.list from. Otherwise, supply vector of species names to species.

spp.list <- createSpp.list(species = NULL, 
                           taxo.dat = NULL, 
                           spp.list_src = spp.list_src)
species list extracted from dataset: D0
 file.name: brainmain2.csv
str(spp.list, vec.len = 3)
'data.frame':   1906 obs. of  4 variables:
 $ species    : chr  "Aix_galericulata" "Aix_sponsa" "Alopochen_aegyptiaca" ...
 $ master.spp : logi  TRUE TRUE TRUE TRUE ...
 $ rel.spp    : logi  NA NA NA NA ...
 $ taxo.status: chr  "original" "original" "original" ...
 - attr(*, "type")= chr "spp.list"

create master shell

master <- create_master(spp.list)
str(master, max.level = 2, vec.len = 3)
List of 3
 $ data    :'data.frame':   0 obs. of  11 variables:
  ..$ species    : logi(0) 
  ..$ synonyms   : logi(0) 
  ..$ data.status: logi(0) 
  ..$ var        : logi(0) 
  ..$ value      : logi(0) 
  ..$ data.ID    : logi(0) 
  ..$ qc         : logi(0) 
  ..$ observer   : logi(0) 
  ..$ ref        : logi(0) 
  ..$ n          : logi(0) 
  ..$ notes      : logi(0) 
  ..- attr(*, "format")= chr "master"
 $ spp.list:'data.frame':   1906 obs. of  4 variables:
  ..$ species    : chr [1:1906] "Aix_galericulata" "Aix_sponsa" "Alopochen_aegyptiaca" ...
  ..$ master.spp : logi [1:1906] TRUE TRUE TRUE TRUE ...
  ..$ rel.spp    : logi [1:1906] NA NA NA NA ...
  ..$ taxo.status: chr [1:1906] "original" "original" "original" ...
  ..- attr(*, "type")= chr "spp.list"
 $ metadata:'data.frame':   38 obs. of  7 variables:
  ..$ code  : chr [1:38] "species" "brain.volume" "brain.mass" ...
  ..$ cat   : chr [1:38] "NOMINAL" "MORPHOLOGICAL" "MORPHOLOGICAL" ...
  ..$ descr : chr [1:38] "Scientific name" "Brain volume" "Brain mass" ...
  ..$ scores: chr [1:38] NA NA NA ...
  ..$ levels: chr [1:38] NA NA NA ...
  ..$ type  : chr [1:38] NA "CON" "CON" ...
  ..$ units : chr [1:38] NA "mm" "g" ...
 - attr(*, "type")= chr "master"
 - attr(*, "file.names")= chr "empty"



match new datasets to master

assign filename to add and create match object from it.

The fuction loads the file specified by filename in {input.folder}pre/csv/. The argument sub specified which of the two sets of species to be matched (spp.list or data) is a subset (ie smaller) than the other. spp.list is the same species attached to the master.

filename <- file.names[file.names == "Amniote_Database_Aug_2015.csv"]

m <- matchObj(data.ID = NULL,
              spp.list = spp.list,
              data = NULL,
              sub = "spp.list", file.name = filename, 
              meta = createMeta(meta.vars)) # use addMeta function to manually add metadata.
List of 7
 $ data.ID  : chr "D1"
 $ spp.list :'data.frame':  1906 obs. of  4 variables:
  ..- attr(*, "type")= chr "spp.list"
 $ data     :'data.frame':  9802 obs. of  20 variables:
  ..- attr(*, "format")= chr "data:wide"
 $ sub      : chr "spp.list"
 $ set      : chr "data"
 $ meta     :List of 5
  ..- attr(*, "type")= chr "meta"
 $ file.name: Named chr "Amniote_Database_Aug_2015.csv"
  ..- attr(*, "names")= chr "x1"
 - attr(*, "type")= chr "match.object"
 - attr(*, "status")= chr "unmatched"

compile metadata and prepare m for matching

m <- m %>% 
  separateDatMeta() %>% 
  compileMeta(input.folder = input.folder) %>%
  checkVarMeta(master$metadata) %>%
  dataMatchPrep()
[1] "============================================================================"
[1] "processing meta.var: qc"
[1] "Warning: NULL data for meta.var: qc"
[1] "============================================================================"
[1] "processing meta.var: observer"
[1] "Warning: NULL data for meta.var: observer"
[1] "============================================================================"
[1] "processing meta.var: ref"
[1] "loading 'ref/Amniote_Database_Aug_2015.csv'"
[1] "============================================================================"
[1] "processing meta.var: n"
[1] "loading 'n/Amniote_Database_Aug_2015.csv'"
[1] "n vars matched successfully to post/n/Amniote_Database_Aug_2015_n_group.csv"
[1] "============================================================================"
[1] "processing meta.var: notes"
[1] "Warning: NULL data for meta.var: notes"
[1] "D1 metadata complete"
List of 7
 $ data.ID  : chr "D1"
 $ spp.list :'data.frame':  1906 obs. of  4 variables:
  ..- attr(*, "type")= chr "spp.list"
 $ data     :'data.frame':  9802 obs. of  22 variables:
 $ sub      : chr "spp.list"
 $ set      : chr "data"
 $ meta     :List of 5
  ..- attr(*, "type")= chr "meta"
 $ file.name: Named chr "Amniote_Database_Aug_2015.csv"
  ..- attr(*, "names")= chr "x1"
 - attr(*, "type")= chr "match.object"
 - attr(*, "status")= chr "unmatched"

match m to spp.list

m <- dataSppMatch(m, syn.links = syn.links, addSpp = T)
Warning in dataSppMatch(m, syn.links = syn.links, addSpp = T): match
incomplete, 8 spp.list datapoints unmatched
List of 8
 $ data.ID  : chr "D1"
 $ spp.list :'data.frame':  1906 obs. of  4 variables:
  ..- attr(*, "type")= chr "spp.list"
 $ data     :'data.frame':  1898 obs. of  22 variables:
  ..- attr(*, "format")= chr "data:wide"
 $ sub      : chr "spp.list"
 $ set      : chr "data"
 $ meta     :List of 5
  ..- attr(*, "type")= chr "meta"
 $ file.name: Named chr "Amniote_Database_Aug_2015.csv"
  ..- attr(*, "names")= chr "x1"
 $ unmatched:'data.frame':  8 obs. of  2 variables:
 - attr(*, "type")= chr "match.object"
 - attr(*, "status")= chr "matched:incomplete - 8 spp.list unmatched"

output <- masterDataFormat(m, meta.vars, match.vars, var.vars)
Warning in masterDataFormat(m, meta.vars, match.vars, var.vars): 1575 data points missing reference information!:
 (9.6% of 16469)
str(output, max.level = 1, vec.len = 3)
List of 3
 $ data     :'data.frame':  16469 obs. of  11 variables:
  ..- attr(*, "format")= chr "master"
 $ spp.list :'data.frame':  1906 obs. of  4 variables:
  ..- attr(*, "type")= chr "spp.list"
 $ file.name: Named chr "Amniote_Database_Aug_2015.csv"
  ..- attr(*, "names")= chr "D1"
 - attr(*, "type")= chr "data:master"
 - attr(*, "status")= chr "matched"

merge to master

master <- updateMaster(master, output = output)
str(master, max.level = 1, vec.len = 3)
List of 3
 $ data    :'data.frame':   16469 obs. of  11 variables:
  ..- attr(*, "format")= chr "master"
 $ spp.list:'data.frame':   1906 obs. of  4 variables:
  ..- attr(*, "type")= chr "spp.list"
 $ metadata:'data.frame':   38 obs. of  7 variables:
 - attr(*, "type")= chr "master"
 - attr(*, "file.names")= Named chr "Amniote_Database_Aug_2015.csv"
  ..- attr(*, "names")= chr "D1"


interactive view of framework objects:

master

jsonedit(master)


matched m

jsonedit(m)
LS0tCnRpdGxlOiAicm1hY3JvUkRNX3dvcmtmbG93IgphdXRob3I6ICJBbm5hIEtyeXN0YWxsaSIKZGF0ZTogIjE3IFNlcHRlbWJlciAyMDE2IgpvdXRwdXQ6IAogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQogICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCiAgICB0aGVtZTogcGFwZXIKICAgIHNlbGZfY29udGFpbmVkOiBmYWxzZQogICAgCiAgICAKLS0tCiMgc2V0dXAKCmBgYHtyIGdsb2JhbC1zZXR1cCwgZWNobyA9IEZ9CnJtKGxpc3Q9bHMoKSkKb3B0aW9ucyhzdHJpbmdzQXNGYWN0b3JzID0gRikKYGBgCgpgYGB7ciBybWQtc2V0dXAsIGVjaG89RkFMU0UsIHB1cmw9RkFMU0V9CnJlcXVpcmUoUkN1cmwpCnJlcXVpcmUoa25pdHIpCmxpYnJhcnkobGlzdHZpZXdlcikKa25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFKQpgYGAKCgojIyBzb3VyY2Ugcm1hY3JvUkRNIGZ1bmN0aW9ucwoKRmlyc3Qgc291cmNlIHRoZSBybWFjcm9SRE0gZnVuY3Rpb25zLiBDdXJyZW50bHkgdGhlIGJlc3Qgd2F5IGlzIHRvIGp1c3Qgc291cmNlIGZyb20gZ2l0aHViIHVzaW5nIGBSQ3VscjpnZXRVUkwoKWAuCgoqKipXQVJOSU5HOiBVbmRlciBjb250aW51b3VzIGRldmVsb3BtZW50KioqCgpgYGB7ciBzb3VyY2Utcm1hY3JvUkRNLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpyZXF1aXJlKFJDdXJsKQpldmFsKHBhcnNlKHRleHQgPSBnZXRVUkwoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9hbm5ha3J5c3RhbGxpL3JtYWNyb1JETS9tYXN0ZXIvUi9mdW5jdGlvbnMuUiIsIHNzbC52ZXJpZnlwZWVyID0gRkFMU0UpKSkKZXZhbChwYXJzZSh0ZXh0ID0gZ2V0VVJMKCJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vYW5uYWtyeXN0YWxsaS9ybWFjcm9SRE0vbWFzdGVyL1Ivd2lkZURhdGFfZnVuY3Rpb24uUiIsIHNzbC52ZXJpZnlwZWVyID0gRkFMU0UpKSkKCmBgYAoKIyMgc2V0dXAgZmlsZS5zeXN0ZW0KCk5leHQsIHRvIGluaXRpYWxpc2UgdGhlIHByb2plY3Qgd2UgbmVlZCB0byBzdXBwbHkgdmFsaWQgcGF0aHdheXMgdG8gcHJvamVjdCBmb2xkZXJzIGNvbnRhaW5pbmc6CgotICoqZGF0YVw6KiogZm9sZGVyIGNvbnRhaW5nIHByaXZhdGUgaW5wdXQgYW5kIG91dHB1dCBmb2xkZXJzIGFyZSBzdG9yZWQsIHVzdWFsbHkgYSBnb29nbGVkcml2ZSBmb2xkZXIpIGFuZCAKLSAqKmNvZGU6KiogZm9sZGVyIGNvbnRhaW5pbmcgc2NyaXB0cyBhc3NvY2lhdGVkIHdpdGggdGhlIHByb2plY3QuIFRoaXMgaXMgdXN1YWxseSBhbiBSU3R1ZGlvIHByb2plY3QgZGlyZWN0b3J5LCBpZGVhbGx5IHZlcnNpb24gY29udHJvbGxlZCBvbiBnaXRodWIuCgpUaGUgZnVuY3Rpb24gc2V0cyB0aG9zZSBkaXJlY3RvcmllcyBpbiB0aGUgZ2xvYmFsIGVudmlyb25tZW50LgoKYGBge3Igc2V0LWRpcnN9CgpzZXREaXJlY3RvcmllcyhzY3JpcHQuZm9sZGVyID0gIn4vRG9jdW1lbnRzL3dvcmtmbG93cy9CcmFpbl9zaXplX2V2b2x1dGlvbi8iLCAKICAgICAgICAgICAgICAgZGF0YS5mb2xkZXIgPSAifi9Hb29nbGUgRHJpdmUvQnJhaW4gZXZvbHV0aW9uLyIsCiAgICAgICAgICAgICAgIGVudmlyID0gZ2xvYmFsZW52KCkpCgpgYGAKCk9uY2UgcHJvamVjdCBmb2xkZXJzIGhhdmUgYmVlbiBzZXQsIHdlIHNldCB1cCB0aGUgKmZpbGUgc3lzdGVtKiBieSBjcmVhdGluZyB0aGUgcmVxdWlyZWQgZm9sZGVycyAoaWYgdGhleSBkb24ndCBleGlzdCBhbHJlYWR5KSBpbiB0aGUgcHJvamVjdCBmb2xkZXJzLiAKCmBgYHtyIHNldHVwLWZpbGUuc3lzdGVtfQoKc2V0dXBGaWxlU3lzdGVtKHNjcmlwdC5mb2xkZXIgPSAifi9Eb2N1bWVudHMvd29ya2Zsb3dzL0JyYWluX3NpemVfZXZvbHV0aW9uLyIsIAogICAgICAgICAgICAgICAgZGF0YS5mb2xkZXIgPSAifi9Hb29nbGUgRHJpdmUvQnJhaW4gZXZvbHV0aW9uLyIpCmBgYAoKIyMgaW5pdGlhbGlzZSBkYXRhYmFzZSBjb25maWd1cmF0aW9ucyBhbmQgYXR0YWNoIHRvIHNlYXJjaCBwYXRod2F5CgpJbiB0aGlzIHN0ZXAsIHdlIGluaXRpYWxpc2UgdGhlIGVudmlyb25tZW50IHdpdGggc29tZSByZXF1aXJlZCBwYXJhbWV0ZXJzIHRvIGJ1aWxkIHRoZSBkYXRhYmFzZSBhbmQgcHJvY2VzcyBmaWxlcyB0byBpdC4gVGhlIGNhbGwgYmVsb3cgc2hvd3MgdGhlIGRlZmF1bHQgaW5pdGlhbGlzYXRpb24gc2V0dGluZ3Mgd2hpY2ggeW91IHdvdWxkIGdldCBpZiB5b3UganVzdCBjYWxsZWQgYGludGlfZGIoKWAKCmBgYHtyIG1hc3Rlci1jb25maWd1cmF0aW9uLWRlZmF1bHQsIGV2YWw9RiwgcHVybD1GQUxTRX0KaW5pdF9kYih2YXIudmFycyA9IGMoInZhciIsICJ2YWx1ZSIsICJkYXRhLklEIiksCiAgICAgICAgICAgICAgICAgICAgbWF0Y2gudmFycyA9IGMoInN5bm9ueW1zIiwgImRhdGEuc3RhdHVzIiksCiAgICAgICAgICAgICAgICAgICAgbWV0YS52YXJzID0gYygicWMiLCAib2JzZXJ2ZXIiLCAicmVmIiwgIm4iLCAibm90ZXMiKSwKICAgICAgICAgICAgICAgICAgICB0YXhvLnZhcnMgPSBjKCJnZW51cyIsICJmYW1pbHkiLCAib3JkZXIiKSwKICAgICAgICAgICAgICAgICAgICBzcHAubGlzdF9zcmMgPSBOVUxMKQpgYGAKCgpgYGB7ciBtYXN0ZXItY29uZmlndXJhdGlvbiwgZXZhbD1UfQppbml0X2RiKHNwcC5saXN0X3NyYyA9ICJEMCIpCmBgYAoKVGhlIGZ1bmN0aW9uIGFwcGVuZHMgdGhlIG9iamVjdHMgc3BlY2lmaWVkIGluIHRoZSBmdW5jdGlvbiB0byBlbnZpcm9ubWVudCBgbWFzdGVyX2NvbmZpZ2AgYXQgcG9zaXRpb24gMiBpbiB0aGUgc2VhcmNoIHBhdGggKG5vdGUgcG9zaXRpb24gb2YgYEdsb2JhbEVudmlyb25tZW50YCA9IDEpLiAKCkhlcmUncyBhIGxpc3Qgb2YgdGhlIHZhbHVlcyBvZiB0aGUgb2JqZWN0cyB3ZSBqdXN0IGF0dGFjaGVkIGFzIGNvbmZpZ3VyYXRpb25zOgpgYGB7ciBwcmludC1tYXN0ZXJfY29uZmlnLCBlY2hvPUZBTFNFLCBwdXJsPUZBTFNFfQptc19jb25mIDwtIHNldE5hbWVzKGxhcHBseShscygibWFzdGVyX2NvbmZpZyIpLCBGVU4gPSBnZXQsIGVudmlyID0gZW52aXJvbm1lbnQoKSksIAogICAgICAgICBscygibWFzdGVyX2NvbmZpZyIpKQpqc29uZWRpdChtc19jb25mKQpgYGAKCgojIyBzZXR1cCBpbnB1dC5mb2xkZXIKCk5leHQgd2Ugc2V0dXAgdGhlIGZvbGRlcnMgaW4gYGlucHV0LmZvbGRlci9wcmUvYCBhbmQgYHBvc3QvYCBhY2NvcmRpbmcgdG8gdGhlIGNvbmZpZ3VyYXRpb25zIHNldCBpbiB0aGUgcHJldmlvdXMgc3RlcC4KCklmIHRoZSBjb3JyZWN0IHNldHVwIGFscmVhZHkgZXhpc3RzLCBubyBhY3Rpb24gaXMgdGFrZW46CgpgYGB7ciBzZXR1cC1pbnB1dC5mb2xkZXJ9CnNldHVwSW5wdXRGb2xkZXIoaW5wdXQuZm9sZGVyKQpgYGAKCgpgYGB7ciBwcmludC1pYS1mb2xkZXItc3RyLCBwdXJsPUZBTFNFfQpkaXJzIDwtIGxpc3QuZGlycyhwYXN0ZShpbnB1dC5mb2xkZXIsICJwcmUvIiwgc2VwID0gIiIpLCBmdWxsLm5hbWVzID0gVCkKCnByaW50KGRpcnMpCmBgYAoKPGJyPgoKKioqCgojIHBvcHVsYXRlIGZpbGUuc3lzdGVtCgpUaGUgZnVuY3Rpb25zIHRha2UgYWR2YW50YWdlIG9mIHRoZSBzdHJ1Y3R1cmUgb2YgdGhlIGZpbGUuc3l0ZW0gdG8gYXV0b21hdGUgbG9hZGluZyBhbmQgbGlua2luZyBvZiBkYXRhIGFuZCBtZXRhZGF0YSB0aHJvdWdoIGFwcHJvcHJpYXRlIG5hbWluZyBhbmQgbG9jYXRpb24gb2YgZmlsZXMgd2l0aGluIHRoZSBmaWxlLnN5c3RlbS4KCi0gKipgcmF3L2AqKiBvcmdhbmlzZSBhbGwgcmF3IGRhdGEKLSAqKmBwcmUvYCBzYXZlIGNvcGllcyBvZiB0aGUgcmF3IGRhdGEgZmlsZXMgaW4gdGhlIGFwcHJvcHJpYXRlIGRhdGEgKGBjdnNgKSBvciBgbWV0YS52YXJzYCBmb2xkZXJzLioqCgoqKipOQioqKiAqYG1ldGEudmFyYCBkYXRhIHNoZWV0cyBzaG91bGQgYmUgbmFtZWQgd2l0aCB0aGUgc2FtZSBuYW1lIGFzIHRoZSBgZGF0YWAgZGF0YSBzaGVldC4gVGFrZSBjYXJlIGR1cmluZyB0aGlzIHN0YWdlIHRvIGVuc3VyZSBmaWxlcyBhcmUgbmFtZWQgY29ycmVjdGx5IGFuZCBzdG9yZWQgaW4gdGhlIGFwcHJvcHJpYXRlIGZvbGRlcnMuKiAKCiMgbG9hZCByZXF1aXJlZCBkYXRhCgojIyBtYWtlIGZjb2RlcyAoZm9sZGVyIGNvZGVzKSB2ZWN0b3IKClRoZSBgZmNvZGVzYCB2ZWN0b3Igc3BlY2lmaWVzIHRoZSBkZXRhaWxzIG9mIGZvbGRlcnMgaW4gYHByZS9gIGFuZCBgcG9zdC9gIGBpbnB1dC5mb2xkZXJgIGZvbGRlcnMuIEl0IGFsc28gY3JlYXRlcyBhcHByb3ByaWF0ZSBjb2RlIHByZWZpeGVzIGZvciBlYWNoIHR5cGUgb2YgYGRhdGFgIG9yIGBtZXRhLnZhcmAgc2hlZXQuIE5vdGUgdGhhdCAiRCIgaXMgcmVzZXJ2ZWQgZm9yIGBkYXRhYCBmaWxlcywgIlIiIGZvciBgcmVmYCBmaWxlcyBhbmQgIk4iIGZvciBgbmAgZmlsZXMuCgpgYGB7ciBlbnN1cmUtZmNvZGVzfQpmY29kZXMgPC0gZW5zdXJlX2Zjb2RlcyhtZXRhLnZhcnMpCgpgYGAKCmBgYHtyIHByaW50LWZjb2RlcywgcHVybD1GQUxTRX0KcHJpbnQoZmNvZGVzKQpgYGAKCgojIyBzZXQgZmlsZS5uYW1lcyB2ZWN0b3IKClNwZWNpZnkgdGhlIGBmaWxlLm5hbWVzYCBvZiB0aGUgZmlsZXMgeW91IHdpc2ggdG8gcHJvY2Vzcy4gSWYgeW91IHdhbnQgdG8gcHJvY2VzcyBhbGwgZmlsZXMgaW4gdGhlIGZpbGUuc3lzdGVtIHVzZSBgZmlsZS5uYW1lcyA9IE5VTExgLiAKCmBgYHtyIHNldC1maWxlLm5hbWVzfQpmaWxlLm5hbWVzIDwtIGNyZWF0ZV9maWxlLm5hbWVzKGZpbGUubmFtZXMgPSBjKCJicmFpbm1haW4yLmNzdiIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIkFtbmlvdGVfRGF0YWJhc2VfQXVnXzIwMTUuY3N2IiwgImFuYWdlZGF0YXNldGYuY3N2IikpCgoKYGBgCgpgYGB7ciBwcmludC1maWxlLm5hbWVzLCBwdXJsPUZBTFNFfQpwcmludChmaWxlLm5hbWVzKQpgYGAKCgojIyBsb2FkIHN5c3RlbSByZWZlcmVuY2UgZmlsZXMKCmBgYHtyIGxvYWQtc3lzLnJlZn0KbG9hZF9zeXMucmVmKGZpbGVFbmNvZGluZyA9ICJtYWMiLCB2aWV3ID0gRikKYGBgCip1c2UgYHZpZXcgPSBUYCB0byBvcGVuIGEgdmlld2VyIGZvciBlYWNoIG9mIHRoZSBzeXMucmVmIGZpbGVzCgojIyMgYG1ldGFkYXRhLmNzdmAKCmBgYHtyIHRhYmxlLW1ldGFkYXRhLCBwdXJsPUZBTFNFfQoKa2FibGUoaGVhZChtZXRhZGF0YSwgMTApKQpgYGAKCgojIyMgYGRhdGFfbG9nLmNzdmAKCmBgYHtyIHRhYmxlLWRhdGFfbG9nLCBwdXJsPUZBTFNFfQoKCmthYmxlKGRhdGFfbG9nKQpgYGAKCiMjIyBgdm5hbWVzLmNzdmAKCmBgYHtyIHRhYmxlLXZuYW1lcywgcHVybD1GQUxTRX0KCmthYmxlKGhlYWQodm5hbWVzLCAxMCkpCmBgYAoKIyMgbG9hZCBgc3luLmxpbmtzLmNzdmAKClVzZWQgZm9yIHRheG9ub21pYyBtYXRjaGluZyAocGxhbnMgdG8gYXV0b21hdGUgdGhpcyBieSBpbnRlZ3JhdGluZyBwYWNrYWdlIGB0YXhpemVgLiBzdXBwbGllZCBzeW4ubGlua3Mgb25seSByZWxhdGVzIHRvIGJpcmRzKS4KCmBgYHtyIGxvYWQtc3luLmxpbmtzfQpzeW4ubGlua3MgPC0gcmVhZC5jc3YodGV4dD1nZXRVUkwoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9hbm5ha3J5c3RhbGxpL3JtYWNyb1JETS9tYXN0ZXIvZGF0YS9pbnB1dC90YXhvL3N5bi5saW5rcy5jc3YiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNzbC52ZXJpZnlwZWVyID0gRkFMU0UpLCBoZWFkZXI9VCkKYGBgCgo8YnI+CgoqKioKCiMgcHJvY2VzcyBmaWxlLnN5c3RlbQoKVGhpcyBzdGVwIHByb2Nlc3NlcyB0aGUgLmNzdiBjb3BpZXMgb2YgdGhlIHJhdyBkYXRhIGluIHRoZSBgcHJlL2AgZm9sZGVyIGFuZCB3cml0ZXMgdGhlIHByb2Nlc3NlZCBmaWxlcyBhcyBjc3ZzIHRvIHRoZSBgcG9zdC9gIGZvbGRlciwgcmVhZHkgdG8gYmUgbWF0Y2hlZC4gVGhlIHByb2Nlc3NpbmcgY29uc2VydmVzIHRoZSBmaWxlLm5hbWVzIHRocm91Z2hvdXQuIEl0IGlzIGltcG9ydGFudCBmb3IgdGhpcyBzdGVwIHRoYXQgdGhlICpmaWxlLnN5c3RlbSogaXMgY29ycmVjdGx5IHBvcHVsYXRlZCAoaWUuIGBtZXRhLnZhcmAgZGF0YSBzaGVldHMgc2hvdWxkIGJlIG5hbWVkIHdpdGggdGhlIHNhbWUgbmFtZSBhcyB0aGUgYGRhdGFgIGRhdGEgc2hlZXQgYW5kIGluIHRoZSBjb3JyZWN0IGZvbGRlcikuIAoKCgo8YnI+CgoqKlRoZSBmdW5jdGlvbiBydW5zIGEgYmFzaWMgcHJvY2Vzc2luZyBzdGFnZSBmb3IgZWFjaCBmaWxlIGluIGBmaWxlLm5hbWVzYDoqKgoKLSBwcm9jZXNzaW5nIGlzIGl0ZXJhdGVkIG92ZXI6CiAgICAtICoqYWxsIGZpbGUubmFtZXMqKiBhdmFpbGFibGUgaW4gdGhlIGZpbGUuc3lzdGVtIGlmIGBmaWxlLnN5c3RlbSA9ICJmcm9tRlMiYCwKICAgIC0gKipmaWxlcyBzcGVjaWZpZWQgaW4gYGZpbGUubmFtZXNgKiogaWYgYGZpbGUubmFtZXNgIGlzIHZlY3RvciBvZiBmaWxlIG5hbWVzIChub3RlIHRoYXQgb25seSBmaWxlcyBhdmFpbGFibGUgaW4gdGhlIGZpbGUuc3lzdGVtIGFyZSBwcm9jZXNzZWQpLgotIGRhdGEgYXJlIGxvYWRlZCwgdHJpbW1lZCBvZiB3aGl0ZXNwYWNlLCBibGFuayBsaW5lcyBhbmQgYGMoIiIsICIgIiwgIk5BIiwgIi05OTkiKWAgY29kZWQgYXMgYE5BYHMgYnkgZGVmYXVsdC4KLSBjb2x1bW4gbmFtZXMgaXMgdGhlIGRhdGEgYXJlIG1hdGNoZWQgdG8gbWFzdGVyIHZhcmlhYmxlIGNvZGVzIHRocm91Z2ggYHZuYW1lc2AKLSBpZiB0aGUgb3JpZ2luYWwgZGF0YXNldCBjb250YWlucyBzcGVjaWVzIGRldGFpbHMgYWNyb3NzIHR3byBjb2x1bW5zICgqc3BlY2llcyogYW5kICpnZW51cyopLCBkYXRhIGluIHRoZSBjb2x1bW5zIGFyZSBjb25jYXRpbmF0ZWQgaW4gdGhlIGZvcm0gYCJnZW51c19zcGVjaWVzImAgYW5kIG1lcmdlZCBpbnRvIGEgc2luZ2xlIGAkc3BlY2llc2AgY29sdW1uLiAqKiplbnN1cmUgZ2VudXMgY29sdW1uIG5hbWUgaW4gZmlsZXMgaXMgbWF0Y2hlZCB0byBgImdlbnVzImAgaW4gYHZuYW1lc2AuKioqIAoKPGJyPgoKKipjdXN0b20gcHJvY2Vzc2luZyBzY3JpcHRzKioKCnlvdSBjYW4gaW5jbHVkZSBleHRyYSBwcm9jZXNzaW5nIHNjcmlwdHMgZm9yIGluZGl2aWR1YWwgZmlsZXMgYnkgYWRkaW5nIHRoZW0gaW50byB0aGUgYHtzY3JpcHQuZm9sZGVyfXByb2Nlc3MvYCBmb2xkZXIuIFRvIGJlIGxvYWRlZCBjb3JyZWN0bHksICoqc2NyaXB0cyBuZWVkIHRvIGJlIG5hbWVkIGFwcHJvcHJpYXRlbHk6KioKCi0gdG8gc291cmNlIGZvciBhY3Jvc3MgKmFsbCBmaWxlcyogaW4gZmlsZS5zeXN0ZW0gbWF0Y2hpbmcgdGhlIGBmaWxlLm5hbWVgOiBuYW1lIHNjcmlwdCBhcyBgZmlsZS5uYW1lYC4gZWcgLmAiQW1uaW90ZV9EYXRhYmFzZV9BdWdfMjAxNS5SImAKLSB0byBzb3VyY2UgZm9yICppbmRpdmlkdWFsIGZpbGVzKiBpbiBmaWxlLnN5c3RlbSBtYXRjaGluZyB0aGUgYGZpbGUubmFtZWA6IG5hbWUgc2NyaXB0IGFzIGBmaWxlLm5hbWVgIGFwcGVuZCB3aXRoIGFwcHJvcHJpYXRlIGBmY29kZWAsIGVnIC5gIkFtbmlvdGVfRGF0YWJhc2VfQXVnXzIwMTVfcmVmLlIiYAoKCmBgYHtyIHByb2Nlc3MtY3N2cywgd2FybmluZz1GQUxTRX0KCnByb2Nlc3NfZmlsZS5zeXN0ZW0oZmlsZS5uYW1lcywgZmNvZGVzKQoKYGBgCgo8YnI+CgoqKioKCiMgQ3JlYXRlIGRhdGFiYXNlCgojIyBjcmVhdGUgc3BwLmxpc3QKClVzZSBgc3BwLmxpc3Rfc291cmNlYCB0byBzcGVjaWZ5IGRjb2RlIG9mIGZpbGUubmFtZSB0byBleHRyYWN0IHNwcC5saXN0IGZyb20uIE90aGVyd2lzZSwgCnN1cHBseSB2ZWN0b3Igb2Ygc3BlY2llcyBuYW1lcyB0byBgc3BlY2llc2AuCgpgYGB7ciBjcmVhdGUtc3BwLmxpc3R9CnNwcC5saXN0IDwtIGNyZWF0ZVNwcC5saXN0KHNwZWNpZXMgPSBOVUxMLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgdGF4by5kYXQgPSBOVUxMLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgc3BwLmxpc3Rfc3JjID0gc3BwLmxpc3Rfc3JjKQpzdHIoc3BwLmxpc3QsIHZlYy5sZW4gPSAzKQoKYGBgCgojIyBjcmVhdGUgbWFzdGVyIHNoZWxsCgpgYGB7ciBjcmVhdGUtbWFzdGVyfQoKCm1hc3RlciA8LSBjcmVhdGVfbWFzdGVyKHNwcC5saXN0KQpzdHIobWFzdGVyLCBtYXgubGV2ZWwgPSAyLCB2ZWMubGVuID0gMykKYGBgCgo8YnI+CgoqKioKCiMgbWF0Y2ggbmV3IGRhdGFzZXRzIHRvIG1hc3RlcgoKIyMgYXNzaWduIGZpbGVuYW1lIHRvIGFkZCBhbmQgY3JlYXRlIG1hdGNoIG9iamVjdCBmcm9tIGl0LiAKVGhlIGZ1Y3Rpb24gbG9hZHMgdGhlIGZpbGUgc3BlY2lmaWVkIGJ5IGBmaWxlbmFtZWAgaW4gYHtpbnB1dC5mb2xkZXJ9cHJlL2Nzdi9gLiBUaGUgYXJndW1lbnQgYHN1YmAgc3BlY2lmaWVkIHdoaWNoIG9mIHRoZSB0d28gc2V0cyBvZiBzcGVjaWVzIHRvIGJlIG1hdGNoZWQgKGBzcHAubGlzdGAgb3IgYGRhdGFgKSBpcyBhIHN1YnNldCAoaWUgc21hbGxlcikgdGhhbiB0aGUgb3RoZXIuIGBzcHAubGlzdGAgaXMgdGhlIHNhbWUgc3BlY2llcyBhdHRhY2hlZCB0byB0aGUgbWFzdGVyLgpgYGB7ciBjcmVhdGUtbX0KCmZpbGVuYW1lIDwtIGZpbGUubmFtZXNbZmlsZS5uYW1lcyA9PSAiQW1uaW90ZV9EYXRhYmFzZV9BdWdfMjAxNS5jc3YiXQoKbSA8LSBtYXRjaE9iaihkYXRhLklEID0gTlVMTCwKICAgICAgICAgICAgICBzcHAubGlzdCA9IHNwcC5saXN0LAogICAgICAgICAgICAgIGRhdGEgPSBOVUxMLAogICAgICAgICAgICAgIHN1YiA9ICJzcHAubGlzdCIsIGZpbGUubmFtZSA9IGZpbGVuYW1lLCAKICAgICAgICAgICAgICBtZXRhID0gY3JlYXRlTWV0YShtZXRhLnZhcnMpKSAjIHVzZSBhZGRNZXRhIGZ1bmN0aW9uIHRvIG1hbnVhbGx5IGFkZCBtZXRhZGF0YS4KCmBgYAoKYGBge3IgcHJpbnQtcHJlbWF0Y2gtbSwgZWNobz1GQUxTRSwgcHVybD1GQUxTRX0Kc3RyKG0sIG1heC5sZXZlbCA9IDEsIHZlYy5sZW4gPSAzKQpgYGAKCiMjIGNvbXBpbGUgbWV0YWRhdGEgYW5kIHByZXBhcmUgbSBmb3IgbWF0Y2hpbmcKCmBgYHtyIHByb2Nlc3MtbX0KbSA8LSBtICU+JSAKICBzZXBhcmF0ZURhdE1ldGEoKSAlPiUgCiAgY29tcGlsZU1ldGEoaW5wdXQuZm9sZGVyID0gaW5wdXQuZm9sZGVyKSAlPiUKICBjaGVja1Zhck1ldGEobWFzdGVyJG1ldGFkYXRhKSAlPiUKICBkYXRhTWF0Y2hQcmVwKCkKYGBgCgoKYGBge3IgcHJpbnQtcHJlbWF0Y2gxLW0sIGVjaG89RkFMU0UsIHB1cmw9Rn0Kc3RyKG0sIG1heC5sZXZlbCA9IDEsIHZlYy5sZW4gPSAzKQpgYGAKCiMjIG1hdGNoIG0gdG8gc3BwLmxpc3QKCmBgYHtyIGRhdGEtc3BwLW1hdGNofQptIDwtIGRhdGFTcHBNYXRjaChtLCBzeW4ubGlua3MgPSBzeW4ubGlua3MsIGFkZFNwcCA9IFQpCmBgYAoKCmBgYHtyIHByaW50LXBvc3RtYXRjaC1tLCBlY2hvPUZBTFNFLCBwdXJsPUZ9CnN0cihtLCBtYXgubGV2ZWwgPSAxLCB2ZWMubGVuID0gMykKYGBgCgojIyMgCgpgYGB7ciBvdXRwdXR9Cm91dHB1dCA8LSBtYXN0ZXJEYXRhRm9ybWF0KG0sIG1ldGEudmFycywgbWF0Y2gudmFycywgdmFyLnZhcnMpCmBgYAoKYGBge3IgcHJpbnQtb3V0cHV0LCBwdXJsPUZ9CnN0cihvdXRwdXQsIG1heC5sZXZlbCA9IDEsIHZlYy5sZW4gPSAzKQpgYGAKCgojIyMgbWVyZ2UgdG8gbWFzdGVyCgpgYGB7ciBtZXJnZS10by1tYXN0ZXJ9Cm1hc3RlciA8LSB1cGRhdGVNYXN0ZXIobWFzdGVyLCBvdXRwdXQgPSBvdXRwdXQpCmBgYAoKYGBge3IgcHJpbnQtbWFzdGVyLCBwdXJsPUZ9CnN0cihtYXN0ZXIsIG1heC5sZXZlbCA9IDEsIHZlYy5sZW4gPSAzKQpgYGAKCgoqKioKPGJyPgoKIyBpbnRlcmFjdGl2ZSB2aWV3IG9mIGZyYW1ld29yayBvYmplY3RzOgoKIyMgbWFzdGVyCmBgYHtyIHByaW50LWlhLW1hc3RlciwgcHVybD1GfQpqc29uZWRpdChtYXN0ZXIpCmBgYAoKPGJyPgoKIyMgbWF0Y2hlZCBtCmBgYHtyIHByaW50LWlhLW0sIHB1cmw9Rn0KanNvbmVkaXQobSkKYGBgCgo=